nvidia-device-plugin: Fix binary path when using MIG #7201
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
surajssd
requested review from
AbelHu,
Devinwong,
anujmaheshwari1,
bravebeaver,
cameronmeissner,
djsly,
ganeshkumarashok,
juan-lee,
junjiezhang1997,
lilypan26,
phealy,
r2k1,
timmy-wright and
zachary-bailey
as code owners
This was referenced Oct 15, 2025
chewi
approved these changes
Oct 15, 2025
Contributor
ganeshkumarashok
left a comment
ganeshkumarashok
left a comment
There was a problem hiding this comment.
Can we add an E2E for checking this? That would have failed if not for this change.
surajssd
force-pushed
the
suraj/fix-device-plugin-path-for-mig
branch
from
825d989 to
f49513a
Compare
Member
Author
PTAL. |
surajssd
force-pushed
the
suraj/fix-device-plugin-path-for-mig
branch
from
f49513a to
eea238b
Compare
surajssd
force-pushed
the
suraj/fix-device-plugin-path-for-mig
branch
from
eea238b to
3429b4f
Compare
surajssd
force-pushed
the
suraj/fix-device-plugin-path-for-mig
branch
from
3429b4f to
81ac586
Compare
surajssd
force-pushed
the
suraj/fix-device-plugin-path-for-mig
branch
from
81ac586 to
dff3b84
Compare
ganeshkumarashok
approved these changes
Oct 17, 2025
surajssd
force-pushed
the
suraj/fix-device-plugin-path-for-mig
branch
from
dff3b84 to
4fea6e1
Compare
Update the ExecStart path from `/usr/local/bin` to `/usr/bin` to match the actual installation location of the nvidia-device-plugin binary. This ensures the service can properly start the plugin with MIG strategy configuration. Signed-off-by: Suraj Deshmukh <[email protected]>
- Add gpuCountExpected parameter to ValidateNodeAdvertisesGPUResources() to validate exact GPU count instead of just checking > 0 - Add gpuCount parameter to ValidateGPUWorkloadSchedulable() to make GPU resource request configurable - Update all test callers to pass expected GPU count of 1 - Improve logging to show actual vs expected GPU counts for better debugging Signed-off-by: Suraj Deshmukh <[email protected]>
- Add Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG to validate MIG functionality - Configure test with Standard_NC24ads_A100_v4 VM size and MIG2g instance profile - Add ValidateMIGModeEnabled validator to check MIG mode is enabled via nvidia-smi - Add ValidateMIGInstancesCreated validator to verify MIG instances are properly created - Test validates device plugin, DCGM exporter, and GPU resource scheduling with MIG Signed-off-by: Suraj Deshmukh <[email protected]>
Modified ValidateNvidiaDCGMExporterScrapeCommonMetric to accept a metric parameter instead of hardcoding DCGM_FI_DEV_GPU_UTIL. Updated test calls to specify the appropriate metric for each scenario: - Use DCGM_FI_DEV_GPU_UTIL for standard GPU tests (Ubuntu 24.04, Ubuntu 22.04, AzureLinux3) - Use DCGM_FI_DEV_GPU_TEMP for MIG-enabled tests (Ubuntu 24.04 MIG) This allows for more flexible validation based on the specific GPU configuration being tested. Signed-off-by: Suraj Deshmukh <[email protected]>
Add configurable SSH connectivity retry mechanism for scenarios where nodes may reboot during provisioning (e.g., MIG-enabled GPU nodes). - Add WaitForSSHAfterReboot config option to enable retry behavior - Implement retry logic with exponential backoff using wait.PollUntilContextTimeout - Add reboot detection for common SSH error patterns - Enable 5-minute retry window for Ubuntu 2404 MIG GPU test - Maintain backward compatibility with existing single-attempt behavior This resolves flaky test failures when GPU driver installation triggers automatic reboots during node provisioning. Signed-off-by: Suraj Deshmukh <[email protected]>
Signed-off-by: Suraj Deshmukh <[email protected]>
surajssd
force-pushed
the
suraj/fix-device-plugin-path-for-mig
branch
from
4fea6e1 to
52b7831
Compare
ganeshkumarashok
approved these changes
Oct 20, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
What this PR does / why we need it:
Update the ExecStart path from
/usr/local/binto/usr/binto match the actual installation location of the nvidia-device-plugin binary. This ensures the service can properly start the plugin with MIG strategy configuration.e2e tests:
ValidateNodeAdvertisesGPUResources()andValidateGPUWorkloadSchedulable()to accept GPU count parameters instead of hardcoded valuesTest_Ubuntu2404_NvidiaDevicePluginRunning_MIGvalidates MIG functionality on Ubuntu 24.04 withStandard_NC24ads_A100_v4VMsValidateMIGModeEnabled()- Verifies MIG mode is enabled via nvidia-smiValidateMIGInstancesCreated()- Confirms MIG instances are properly created with expected profilesValidateNvidiaDCGMExporterScrapeCommonMetric()now accepts metric parameter instead of hardcodingDCGM_FI_DEV_GPU_UTILfor standard GPU tests andDCGM_FI_DEV_GPU_TEMPfor MIG-enabled testsWaitForSSHAfterRebootconfig option enables SSH retry with exponential backoff for scenarios where nodes reboot during provisioningWhich issue(s) this PR fixes:
#7063 (comment)